Construction Zone Sign Detection

ABSTRACT

Methods and systems for detection of a construction zone sign are described. A computing device, configured to control the vehicle, may be configured to receive, from an image-capture device coupled to the computing device, images of a vicinity of the road on which the vehicle is travelling. Also, the computing device may be configured to determine image portions in the images that may depict sides of the road at a predetermined height range. Further, the computing device may be configured to detect a construction zone sign in the image portions, and determine a type of the construction zone sign. Accordingly, the computing device may be configured to modify a control strategy associated with a driving behavior of the vehicle; and control the vehicle based on the modified control strategy.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 13/603,615, filed on Sep. 5, 2012, and entitled “ConstructionZone Sign Detection,” which is herein incorporated by reference as iffully set forth in this description

BACKGROUND

Autonomous vehicles use various computing systems to aid in transportingpassengers from one location to another. Some autonomous vehicles mayrequire some initial input or continuous input from an operator, such asa pilot, driver, or passenger. Other systems, for example autopilotsystems, may be used only when the system has been engaged, whichpermits the operator to switch from a manual mode (where the operatorexercises a high degree of control over the movement of the vehicle) toan autonomous mode (where the vehicle essentially drives itself) tomodes that lie somewhere in between.

SUMMARY

The present application discloses embodiments that relate to detectionof a construction zone sign. In one aspect, the present applicationdescribes a method. The method may comprise receiving, at a computingdevice configured to control a vehicle, from an image-capture devicecoupled to the computing device, one or more images of a vicinity of aroad on which the vehicle is travelling. The method also may comprisedetermining, using the computing device, one or more image portions inthe one or more images. The one or more image portions may depict sidesof the road at a predetermined height range. The method also maycomprise detecting, using the computing device, a construction zone signin the one or more image portions. The method further may comprisedetermining, using the computing device, a type of the construction zonesign in the one or more image portions. The method also may comprisemodifying, using the computing device, a control strategy associatedwith a driving behavior of the vehicle, based on the type of theconstruction zone sign; and further may comprise controlling, using thecomputing device, the vehicle based on the modified control strategy.

In another aspect, the present application describes a non-transitorycomputer readable medium having stored thereon instructions executableby a computing device of a vehicle to cause the computing device toperform functions. The functions may comprise receiving from animage-capture device coupled to the computing device, one or more imagesof a vicinity of a road on which the vehicle is travelling. Thefunctions also may comprise determining one or more image portions inthe one or more images. The one or more image portions may depict sidesof the road at a predetermined height range. The functions further maycomprise detecting a construction zone sign in the one or more imageportions. The functions also may comprise determining a type of theconstruction zone sign in the one or more image portions. The functionsfurther may comprise modifying a control strategy associated with adriving behavior of the vehicle, based on the type of the constructionzone sign. The functions also may comprise controlling the vehicle basedon the modified control strategy.

In still another aspect, the present application describes a controlsystem for a vehicle. The control system may comprise an image-capturedevice. The control system also may comprise a computing device incommunication with the image-capture device. The computing device may beconfigured to receive, from the image-capture device, one or more imagesof a vicinity of a road on which the vehicle is travelling. Thecomputing device also may be configured to determine one or more imageportions in the one or more images. The one or more image portions maydepict sides of the road at a predetermined height range. The computingdevice further may be configured to detect a construction zone sign inthe one or more image portions. The computing device also may beconfigured to determine a type of the construction zone sign in the oneor more image portions. The computing device further may be configuredto modify a control strategy associated with a driving behavior of thevehicle, based on the type of the construction zone sign; and controlthe vehicle based on the modified control strategy.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a simplified block diagram of an example automobile, inaccordance with an example embodiment.

FIG. 2 illustrates an example automobile, in accordance with an exampleembodiment.

FIG. 3 is a flow chart of a method for detection of a construction zoneusing multiple sources of information, in accordance with an exampleembodiment.

FIG. 4 illustrates a vehicle approaching a construction zone, inaccordance with an example embodiment.

FIG. 5 is a flow chart of a method for detection of a construction zonesign, in accordance with an example embodiment.

FIGS. 6A-6B illustrate images of a road and vicinity of the road thevehicle is travelling on, in accordance with an example embodiment.

FIGS. 6C-6D illustrate portions of the images of the road and thevicinity of the road depicting sides of the road at a predeterminedheight range, in accordance with an example embodiment.

FIG. 7 is a flow chart of a method for detection of the constructionzone sign using LIDAR-based information, in accordance with an exampleembodiment.

FIG. 8A illustrates LIDAR-based detection of the construction zone signin an area at a height greater than a threshold height from a surface ofthe road, in accordance with an example embodiment.

FIG. 8B illustrates a LIDAR-based image depicting the area at the heightgreater than the threshold height from the surface of the road, inaccordance with an example embodiment.

FIG. 9 is a flow chart of a method for detection of construction zoneobjects using LIDAR-based information, in accordance with an exampleembodiment.

FIG. 10A illustrates LIDAR-based detection of a construction zone conein an area within a threshold distance from a surface of the road, inaccordance with an example embodiment.

FIG. 10B illustrates a LIDAR-based image depicting the area within thethreshold distance from the surface of the road, in accordance with anexample embodiment.

FIG. 10C illustrates LIDAR-based detection of construction zone conesforming a lane boundary, in accordance with an example embodiment.

FIG. 10D illustrates a LIDAR-based image depicting construction zonecones forming a lane boundary, in accordance with an example embodiment.

FIG. 11 is a schematic illustrating a conceptual partial view of acomputer program, in accordance with an example embodiment.

DETAILED DESCRIPTION

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols identify similarcomponents, unless context dictates otherwise. The illustrative systemand method embodiments described herein are not meant to be limiting. Itmay be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

An autonomous vehicle operating on a road may be configured to rely onmaps for navigation. In some examples, changes due to existence of aconstruction zone on the road may not be reflected in the maps.Therefore, the autonomous vehicle may be configured to detect theconstruction zone and drive through the construction zone safely.

In an example, a computing device, configured to control the vehicle,may be configured to receive, from an image-capture device coupled tothe computing device, one or more images of a vicinity of the road onwhich the vehicle is travelling. Also, the computing device may beconfigured to determine one or more image portions in the one or moreimages that may depict sides of the road at a predetermined heightrange. Further, the computing device may be configured to detect aconstruction zone sign in the one or more image portions, and determinea type of the construction zone sign in the one or more image portions.Accordingly, the computing device may be configured to modify a controlstrategy associated with a driving behavior of the vehicle; and controlthe vehicle based on the modified control strategy.

An example vehicle control system may be implemented in or may take theform of an automobile. Alternatively, a vehicle control system may beimplemented in or take the form of other vehicles, such as cars, trucks,motorcycles, buses, boats, airplanes, helicopters, lawn mowers,recreational vehicles, amusement park vehicles, farm equipment,construction equipment, trams, golf carts, trains, and trolleys. Othervehicles are possible as well.

Further, an example system may take the form of non-transitorycomputer-readable medium, which has program instructions stored thereonthat are executable by at least one processor to provide thefunctionality described herein. An example system may also take the formof an automobile or a subsystem of an automobile that includes such anon-transitory computer-readable medium having such program instructionsstored thereon.

Referring now to the Figures, FIG. 1 is a simplified block diagram of anexample automobile 100, in accordance with an example embodiment.Components coupled to or included in the automobile 100 may include apropulsion system 102, a sensor system 104, a control system 106,peripherals 108, a power supply 110, a computing device 111, and a userinterface 112. The computing device 111 may include a processor 113, anda memory 114. The memory 114 may include instructions 115 executable bythe processor 113, and may also store map data 116. Components of theautomobile 100 may be configured to work in an interconnected fashionwith each other and/or with other components coupled to respectivesystems. For example, the power supply 110 may provide power to all thecomponents of the automobile 100. The computing device 111 may beconfigured to receive information from and control the propulsion system102, the sensor system 104, the control system 106, and the peripherals108. The computing device 111 may be configured to generate a display ofimages on and receive inputs from the user interface 112.

In other examples, the automobile 100 may include more, fewer, ordifferent systems, and each system may include more, fewer, or differentcomponents. Additionally, the systems and components shown may becombined or divided in any number of ways.

The propulsion system 102 may be configured to provide powered motionfor the automobile 100. As shown, the propulsion system 102 includes anengine/motor 118, an energy source 120, a transmission 122, andwheels/tires 124.

The engine/motor 118 may be or include any combination of an internalcombustion engine, an electric motor, a steam engine, and a Stirlingengine. Other motors and engines are possible as well. In some examples,the propulsion system 102 could include multiple types of engines and/ormotors. For instance, a gas-electric hybrid car could include a gasolineengine and an electric motor. Other examples are possible.

The energy source 120 may be a source of energy that powers theengine/motor 118 in full or in part. That is, the engine/motor 118 maybe configured to convert the energy source 120 into mechanical energy.Examples of energy sources 120 include gasoline, diesel, otherpetroleum-based fuels, propane, other compressed gas-based fuels,ethanol, solar panels, batteries, and other sources of electrical power.The energy source(s) 120 could additionally or alternatively include anycombination of fuel tanks, batteries, capacitors, and/or flywheels. Insome examples, the energy source 120 may provide energy for othersystems of the automobile 100 as well.

The transmission 122 may be configured to transmit mechanical power fromthe engine/motor 118 to the wheels/tires 124. To this end, thetransmission 122 may include a gearbox, clutch, differential, driveshafts, and/or other elements. In examples where the transmission 122includes drive shafts, the drive shafts could include one or more axlesthat are configured to be coupled to the wheels/tires 124.

The wheels/tires 124 of automobile 100 could be configured in variousformats, including a unicycle, bicycle/motorcycle, tricycle, orcar/truck four-wheel format. Other wheel/tire formats are possible aswell, such as those including six or more wheels. The wheels/tires 124of automobile 100 may be configured to rotate differentially withrespect to other wheels/tires 124. In some examples, the wheels/tires124 may include at least one wheel that is fixedly attached to thetransmission 122 and at least one tire coupled to a rim of the wheelthat could make contact with the driving surface. The wheels/tires 124may include any combination of metal and rubber, or combination of othermaterials.

The propulsion system 102 may additionally or alternatively includecomponents other than those shown.

The sensor system 104 may include a number of sensors configured tosense information about an environment in which the automobile 100 islocated. As shown, the sensors of the sensor system include a GlobalPositioning System (GPS) module 126, an inertial measurement unit (IMU)128, a radio detection and ranging (RADAR) unit 130, a laser rangefinderand/or light detection and ranging (LIDAR) unit 132, a camera 134, andactuators 136 configured to modify a position and/or orientation of thesensors. The sensor system 104 may include additional sensors as well,including, for example, sensors that monitor internal systems of theautomobile 100 (e.g., an O₂ monitor, a fuel gauge, an engine oiltemperature, etc.). Other sensors are possible as well.

The GPS module 126 may be any sensor configured to estimate a geographiclocation of the automobile 100. To this end, the GPS module 126 mayinclude a transceiver configured to estimate a position of theautomobile 100 with respect to the Earth, based on satellite-basedpositioning data. In an example, the computing device 111 may beconfigured to use the GPS module 126 in combination with the map data116 to estimate a location of a lane boundary on road on which theautomobile 100 may be travelling on. The GPS module 126 may take otherforms as well.

The IMU 128 may be any combination of sensors configured to senseposition and orientation changes of the automobile 100 based on inertialacceleration. In some examples, the combination of sensors may include,for example, accelerometers and gyroscopes. Other combinations ofsensors are possible as well.

The RADAR unit 130 may be considered as an object detection system thatmay be configured to use radio waves to determine characteristics of theobject such as range, altitude, direction, or speed of the object. TheRADAR unit 130 may be configured to transmit pulses of radio waves ormicrowaves that may bounce off any object in a path of the waves. Theobject may return a part of energy of the waves to a receiver (e.g.,dish or antenna), which may be part of the RADAR unit 130 as well. TheRADAR unit 130 also may be configured to perform digital signalprocessing of received signals (bouncing off the object) and may beconfigured to identify the object.

Other systems similar to RADAR have been used in other parts of theelectromagnetic spectrum. One example is LIDAR (light detection andranging), which may be configured to use visible light from lasersrather than radio waves.

The LIDAR unit 132 may include a sensor configured to sense or detectobjects in an environment in which the automobile 100 is located usinglight. Generally, LIDAR is an optical remote sensing technology that canmeasure distance to, or other properties of, a target by illuminatingthe target with light. The light can be any type of electromagneticwaves such as laser. As an example, the LIDAR unit 132 may include alaser source and/or laser scanner configured to emit pulses of laser anda detector configured to receive reflections of the laser. For example,the LIDAR unit 132 may include a laser range finder reflected by arotating mirror, and the laser is scanned around a scene beingdigitized, in one or two dimensions, gathering distance measurements atspecified angle intervals. In examples, the LIDAR unit 132 may includecomponents such as light (e.g., laser) source, scanner and optics,photo-detector and receiver electronics, and position and navigationsystem.

In an example, The LIDAR unit 132 may be configured to use ultraviolet(UV), visible, or infrared light to image objects and can be used with awide range of targets, including non-metallic objects. In one example, anarrow laser beam can be used to map physical features of an object withhigh resolution.

In examples, wavelengths in a range from about 10 micrometers (infrared)to about 250 nm (UV) could be used. Typically light is reflected viabackscattering. Different types of scattering are used for differentLIDAR applications, such as Rayleigh scattering, Mie scattering andRaman scattering, as well as fluorescence. Based on different kinds ofbackscattering, LIDAR can be accordingly called Rayleigh LIDAR, MieLIDAR, Raman LIDAR and Na/Fe/K Fluorescence LIDAR, as examples. Suitablecombinations of wavelengths can allow for remote mapping of objects bylooking for wavelength-dependent changes in intensity of reflectedsignals, for example.

Three-dimensional (3D) imaging can be achieved using both scanning andnon-scanning LIDAR systems. “3D gated viewing laser radar” is an exampleof a non-scanning laser ranging system that applies a pulsed laser and afast gated camera. Imaging LIDAR can also be performed using an array ofhigh speed detectors and a modulation sensitive detectors arraytypically built on single chips using CMOS (complementarymetal-oxide-semiconductor) and hybrid CMOS/CCD (charge-coupled device)fabrication techniques. In these devices, each pixel may be processedlocally by demodulation or gating at high speed such that the array canbe processed to represent an image from a camera. Using this technique,many thousands of pixels may be acquired simultaneously to create a 3Dpoint cloud representing an object or scene being detected by the LIDARunit 132.

A point cloud may include a set of vertices in a 3D coordinate system.These vertices may be defined by X, Y, and Z coordinates, for example,and may represent an external surface of an object. The LIDAR unit 132may be configured to create the point cloud by measuring a large numberof points on the surface of the object, and may output the point cloudas a data file. As the result of a 3D scanning process of the object bythe LIDAR unit 132, the point cloud can be used to identify andvisualize the object.

In one example, the point cloud can be directly rendered to visualizethe object. In another example, the point cloud may be converted topolygon or triangle mesh models through a process that may be referredto as surface reconstruction. Example techniques for converting a pointcloud to a 3D surface may include Delaunay triangulation, alpha shapes,and ball pivoting. These techniques include building a network oftriangles over existing vertices of the point cloud. Other exampletechniques may include converting the point cloud into a volumetricdistance field and reconstructing an implicit surface so defined througha marching cubes algorithm.

The camera 134 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture images of the environment in which theautomobile 100 is located. To this end, the camera may be configured todetect visible light, or may be configured to detect light from otherportions of the spectrum, such as infrared or ultraviolet light. Othertypes of cameras are possible as well. The camera 134 may be atwo-dimensional detector, or may have a three-dimensional spatial range.In some examples, the camera 134 may be, for example, a range detectorconfigured to generate a two-dimensional image indicating a distancefrom the camera 134 to a number of points in the environment. To thisend, the camera 134 may use one or more range detecting techniques. Forexample, the camera 134 may be configured to use a structured lighttechnique in which the automobile 100 illuminates an object in theenvironment with a predetermined light pattern, such as a grid orcheckerboard pattern and uses the camera 134 to detect a reflection ofthe predetermined light pattern off the object. Based on distortions inthe reflected light pattern, the automobile 100 may be configured todetermine the distance to the points on the object. The predeterminedlight pattern may comprise infrared light, or light of anotherwavelength.

The actuators 136 may, for example, be configured to modify a positionand/or orientation of the sensors.

The sensor system 104 may additionally or alternatively includecomponents other than those shown.

The control system 106 may be configured to control operation of theautomobile 100 and its components. To this end, the control system 106may include a steering unit 138, a throttle 140, a brake unit 142, asensor fusion algorithm 144, a computer vision system 146, a navigationor pathing system 148, and an obstacle avoidance system 150.

The steering unit 138 may be any combination of mechanisms configured toadjust the heading or direction of the automobile 100.

The throttle 140 may be any combination of mechanisms configured tocontrol the operating speed and acceleration of the engine/motor 118and, in turn, the speed and acceleration of the automobile 100.

The brake unit 142 may be any combination of mechanisms configured todecelerate the automobile 100. For example, the brake unit 142 may usefriction to slow the wheels/tires 124. As another example, the brakeunit 142 may be configured to be regenerative and convert the kineticenergy of the wheels/tires 124 to electric current. The brake unit 142may take other forms as well.

The sensor fusion algorithm 144 may include an algorithm (or a computerprogram product storing an algorithm) executable by the computing device111, for example. The sensor fusion algorithm 144 may be configured toaccept data from the sensor system 104 as an input. The data mayinclude, for example, data representing information sensed at thesensors of the sensor system 104. The sensor fusion algorithm 144 mayinclude, for example, a Kalman filter, a Bayesian network, or anotheralgorithm. The sensor fusion algorithm 144 further may be configured toprovide various assessments based on the data from the sensor system104, including, for example, evaluations of individual objects and/orfeatures in the environment in which the automobile 100 is located,evaluations of particular situations, and/or evaluations of possibleimpacts based on particular situations. Other assessments are possibleas well

The computer vision system 146 may be any system configured to processand analyze images captured by the camera 134 in order to identifyobjects and/or features in the environment in which the automobile 100is located, including, for example, lane information, traffic signalsand obstacles. To this end, the computer vision system 146 may use anobject recognition algorithm, a Structure from Motion (SFM) algorithm,video tracking, or other computer vision techniques. In some examples,the computer vision system 146 may additionally be configured to map theenvironment, track objects, estimate speed of objects, etc.

The navigation and pathing system 148 may be any system configured todetermine a driving path for the automobile 100. The navigation andpathing system 148 may additionally be configured to update the drivingpath dynamically while the automobile 100 is in operation. In someexamples, the navigation and pathing system 148 may be configured toincorporate data from the sensor fusion algorithm 144, the GPS module126, and one or more predetermined maps so as to determine the drivingpath for the automobile 100.

The obstacle avoidance system 150 may be any system configured toidentify, evaluate, and avoid or otherwise negotiate obstacles in theenvironment in which the automobile 100 is located.

The control system 106 may additionally or alternatively includecomponents other than those shown.

Peripherals 108 may be configured to allow the automobile 100 tointeract with external sensors, other automobiles, and/or a user. Tothis end, the peripherals 108 may include, for example, a wirelesscommunication system 152, a touchscreen 154, a microphone 156, and/or aspeaker 158.

The wireless communication system 152 may be any system configured to bewirelessly coupled to one or more other automobiles, sensors, or otherentities, either directly or via a communication network. To this end,the wireless communication system 152 may include an antenna and achipset for communicating with the other automobiles, sensors, or otherentities either directly or over an air interface. The chipset orwireless communication system 152 in general may be arranged tocommunicate according to one or more other types of wirelesscommunication (e.g., protocols) such as Bluetooth, communicationprotocols described in IEEE 802.11 (including any IEEE 802.11revisions), cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX,or LTE), Zigbee, dedicated short range communications (DSRC), and radiofrequency identification (RFID) communications, among otherpossibilities. The wireless communication system 152 may take otherforms as well.

The touchscreen 154 may be used by a user to input commands to theautomobile 100. To this end, the touchscreen 154 may be configured tosense at least one of a position and a movement of a user's finger viacapacitive sensing, resistance sensing, or a surface acoustic waveprocess, among other possibilities. The touchscreen 154 may be capableof sensing finger movement in a direction parallel or planar to thetouchscreen surface, in a direction normal to the touchscreen surface,or both, and may also be capable of sensing a level of pressure appliedto the touchscreen surface. The touchscreen 154 may be formed of one ormore translucent or transparent insulating layers and one or moretranslucent or transparent conducting layers. The touchscreen 154 maytake other forms as well.

The microphone 156 may be configured to receive audio (e.g., a voicecommand or other audio input) from a user of the automobile 100.Similarly, the speakers 158 may be configured to output audio to theuser of the automobile 100.

The peripherals 108 may additionally or alternatively include componentsother than those shown.

The power supply 110 may be configured to provide power to some or allof the components of the automobile 100. To this end, the power supply110 may include, for example, a rechargeable lithium-ion or lead-acidbattery. In some examples, one or more banks of batteries could beconfigured to provide electrical power. Other power supply materials andconfigurations are possible as well. In some examples, the power supply110 and energy source 120 may be implemented together, as in someall-electric cars.

The processor 113 included in the computing device 111 may comprise oneor more general-purpose processors and/or one or more special-purposeprocessors. To the extent the processor 113 includes more than oneprocessor; such processors could work separately or in combination. Thecomputing device 111 may be configured to control functions of theautomobile 100 based on input received through the user interface 112,for example.

The memory 114, in turn, may comprise one or more volatile and/or one ormore non-volatile storage components, such as optical, magnetic, and/ororganic storage, and the memory 114 may be integrated in whole or inpart with the processor 113. The memory 114 may contain the instructions115 (e.g., program logic) executable by the processor 113 to executevarious automobile functions.

The components of the automobile 100 could be configured to work in aninterconnected fashion with other components within and/or outside theirrespective systems. To this end, the components and systems of theautomobile 100 may be communicatively linked together by a system bus,network, and/or other connection mechanism (not shown).

Further, while each of the components and systems are shown to beintegrated in the automobile 100, in some examples, one or morecomponents or systems may be removably mounted on or otherwise connected(mechanically or electrically) to the automobile 100 using wired orwireless connections.

The automobile 100 may include one or more elements in addition to orinstead of those shown. For example, the automobile 100 may include oneor more additional interfaces and/or power supplies. Other additionalcomponents are possible as well. In these examples, the memory 114 mayfurther include instructions executable by the processor 113 to controland/or communicate with the additional components.

FIG. 2 illustrates an example automobile 200, in accordance with anembodiment. In particular, FIG. 2 shows a Right Side View, Front View,Back View, and Top View of the automobile 200. Although automobile 200is illustrated in FIG. 2 as a car, other examples are possible. Forinstance, the automobile 200 could represent a truck, a van, asemi-trailer truck, a motorcycle, a golf cart, an off-road vehicle, or afarm vehicle, among other examples. As shown, the automobile 200includes a first sensor unit 202, a second sensor unit 204, a thirdsensor unit 206, a wireless communication system 208, and a camera 210.

Each of the first, second, and third sensor units 202-206 may includeany combination of global positioning system sensors, inertialmeasurement units, RADAR units, LIDAR units, cameras, lane detectionsensors, and acoustic sensors. Other types of sensors are possible aswell.

While the first, second, and third sensor units 202 are shown to bemounted in particular locations on the automobile 200, in some examplesthe sensor unit 202 may be mounted elsewhere on the automobile 200,either inside or outside the automobile 200. Further, while only threesensor units are shown, in some examples more or fewer sensor units maybe included in the automobile 200.

In some examples, one or more of the first, second, and third sensorunits 202-206 may include one or more movable mounts on which thesensors may be movably mounted. The movable mount may include, forexample, a rotating platform. Sensors mounted on the rotating platformcould be rotated so that the sensors may obtain information from eachdirection around the automobile 200. Alternatively or additionally, themovable mount may include a tilting platform. Sensors mounted on thetilting platform could be tilted within a particular range of anglesand/or azimuths so that the sensors may obtain information from avariety of angles. The movable mount may take other forms as well.

Further, in some examples, one or more of the first, second, and thirdsensor units 202-206 may include one or more actuators configured toadjust the position and/or orientation of sensors in the sensor unit bymoving the sensors and/or movable mounts. Example actuators includemotors, pneumatic actuators, hydraulic pistons, relays, solenoids, andpiezoelectric actuators. Other actuators are possible as well.

The wireless communication system 208 may be any system configured towirelessly couple to one or more other automobiles, sensors, or otherentities, either directly or via a communication network as describedabove with respect to the wireless communication system 152 in FIG. 1.While the wireless communication system 208 is shown to be positioned ona roof of the automobile 200, in other examples the wirelesscommunication system 208 could be located, fully or in part, elsewhere.

The camera 210 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture images of the environment in which theautomobile 200 is located. To this end, the camera may take any of theforms described above with respect to the camera 134 in FIG. 1. Whilethe camera 210 is shown to be mounted inside a front windshield of theautomobile 200, in other examples the camera 210 may be mountedelsewhere on the automobile 200, either inside or outside the automobile200.

The automobile 200 may include one or more other components in additionto or instead of those shown.

A control system of the automobile 200 may be configured to control theautomobile 200 in accordance with a control strategy from among multiplepossible control strategies. The control system may be configured toreceive information from sensors coupled to the automobile 200 (on oroff the automobile 200), modify the control strategy (and an associateddriving behavior) based on the information, and control the automobile200 in accordance with the modified control strategy. The control systemfurther may be configured to monitor the information received from thesensors, and continuously evaluate driving conditions; and also may beconfigured to modify the control strategy and driving behavior based onchanges in the driving conditions.

FIG. 3 is a flow chart of a method 300 for detection of a constructionzone using multiple sources of information, in accordance with anexample embodiment. FIG. 4 illustrates a vehicle approaching aconstruction zone, in accordance with an embodiment, to illustrate themethod 300. FIGS. 3 and 4 will be described together.

The method 300 may include one or more operations, functions, or actionsas illustrated by one or more of blocks 302-308. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

In addition, for the method 300 and other processes and methodsdisclosed herein, the flowchart shows functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ormemory, for example, such as a storage device including a disk or harddrive. The computer readable medium may include a non-transitorycomputer readable medium, for example, such as computer-readable mediathat stores data for short periods of time like register memory,processor cache and Random Access Memory (RAM). The computer readablemedium may also include non-transitory media or memory, such assecondary or persistent long term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer readable media may also be any other volatile ornon-volatile storage systems. The computer readable medium may beconsidered a computer readable storage medium, a tangible storagedevice, or other article of manufacture, for example.

In addition, for the method 300 and other processes and methodsdisclosed herein, each block in FIG. 3 may represent circuitry that iswired to perform the specific logical functions in the process.

At block 302, the method 300 includes receiving, at a computing deviceconfigured to control a vehicle, from a plurality of sources ofinformation, information relating to detection of a construction zone ona road on which the vehicle is travelling, and each source ofinformation of the plurality of sources of information may be assigned arespective reliability metric indicative of a level of confidence of thedetection of the construction zone based on the information receivedfrom that source of information. The computing device may be onboard thevehicle or may be off-board but in wireless communication with thevehicle, for example. Also, the computing device may be configured tocontrol the vehicle in an autonomous or semi-autonomous operation mode.Further, the computing device may be configured to receive, from sensorscoupled to the vehicle, information associated with, for example,condition of systems and subsystems of the vehicle, driving conditions,road conditions, etc.

FIG. 4 illustrates a vehicle 402 approaching a construction zone on aroad 404. The computing device, configured to control the vehicle 402,may be configured to receive information relating to detection of theconstruction zone. The information may be received from a plurality ofsources. For example, the information may include image-basedinformation received from an image-capture device or camera (e.g., thecamera 134 in FIG. 1, or the camera 210 in FIG. 2) coupled to thecomputing device. In one example, the image-capture device may beonboard the vehicle 402; but, in another example, the image-capturedevice may be off-board (e.g., a given camera coupled to a trafficsignal post). The image-based information, for example, may beindicative of location of one or more static objects with respect to theroad such as construction zone cone(s) 406, construction zone barrel(s)408, construction equipment 410A-B, construction zone signs 412A-B, etc.The construction zone cone(s) 406 is used hereinafter to refer to asingle cone or a group/series of cones. The image-based information alsomay be indicative of other objects related to construction zones such asorange vests and chevrons. The image-based information also may beindicative of road geometry (e.g., curves, lanes, etc.).

In another example, the information relating to the detection of theconstruction zone may include LIDAR-based information received from alight detection and ranging (LIDAR) sensor (e.g., the LIDAR unit 132 inFIG. 1) coupled to the vehicle 402 and in communication with thecomputing device. The LIDAR sensor may be configured to provide athree-dimensional (3D) point cloud of the road 404 and vicinity of theroad 404. The computing device may be configured to identify objects(e.g., the construction zone cone(s) 406, the construction zonebarrel(s) 408, the construction equipment 410A-B, the construction zonesigns 412A-B, etc.) represented by sets of points in the 3D point cloud,for example.

In still another example, the information may include RADAR-basedinformation received from a radio detection and ranging (RADAR) sensor(e.g., the RADAR unit 130 in FIG. 1) coupled to the vehicle 402 and incommunication with the computing device. For example, the RADAR sensormay be configured to emit radio waves and receive back the emitted radiowaves that bounced off objects on the road 404 or in the vicinity of theroad 404. The received signals or RADAR-based information may beindicative of characteristics of an object off of which the radio wavesbounced. The characteristics, for example, may include dimensionalcharacteristics of the object, distance between the object and thevehicle 402, and whether the object is stationary or moving, in additionto speed and direction of motion.

In yet another example, the information may include traffic information.The traffic information may be indicative of behavior of other vehicles,such as vehicles 414A-B, on the road 404. As an example, the trafficinformation may be received from Global Positioning Satellite (GPS)devices coupled to the vehicles 414A-B. GPS information received from arespective GPS device may be indicative of a position of a respectivevehicle including the respective GPS device with respect to the Earth,based on satellite-based positioning data.

In another example, the vehicles 414A-B may be configured to communicatelocation/position and speed information to a road infrastructure device(e.g., a device on a post on the road 404), and the infrastructuredevice may communicate such traffic information to the computing device.This communication may be referred to as vehicle-to-infrastructurecommunication. Vehicle-to-infrastructure communication may includewireless exchange of critical safety and operational data betweenvehicles (e.g., the vehicle 402 and the vehicles 414A-B) and roadinfrastructure, intended to enable a wide range of safety, mobility,road condition, traffic, and environmental information.Vehicle-to-infrastructure communication may apply to all vehicle typesand all roads, and may transform infrastructure equipment into “smartinfrastructure” through incorporation of algorithms that use dataexchanged between vehicles and infrastructure elements to performcalculations, by the computing device coupled to the vehicle 402 forexample, that may recognize high-risk situations in advance, resultingin driver alerts and warnings through specific countermeasures. As anexample, traffic signal systems on the road 404 may be configured tocommunicate signal phase and timing (SPAT) information to the vehicle402 to deliver active traffic information, safety advisories, andwarnings to the vehicle 402 or a driver of the vehicle 402.

In still another example, the traffic information may be received fromdirect vehicle-to-vehicle communication. In this example, owners of thevehicles 402 and 414A-B may be given an option to opt in or out ofsharing information between vehicles. The vehicles 414A-B on the road404 may include devices (e.g., GPS devices coupled to the vehicles414A-B or mobile phones used by drivers of the vehicles 414A-B) that maybe configured to provide trackable signals to the computing deviceconfigured to control the vehicle 402. The computing device may beconfigured to receive the trackable signals and extract trafficinformation and behavior information of the vehicles 414A-B, forexample.

In yet another example the traffic information may be received from atraffic report broadcast (e.g., radio traffic services). In yet stillanother example, the computing device may be configured to receive thetraffic information from on-board or off-board sensors in communicationwith the computing device configured to control the vehicle 402. As anexample, laser-based sensors can provide speed statistics of vehiclespassing through lanes of a highway, and communicate such information tothe computing device.

Based on the traffic information, the computing device may be configuredto estimate nominal speeds and flow of traffic of other vehicles, suchas the vehicles 414A-B, on the road 404. In an example, the computingdevice may be configured to determine a change in nominal speed and flowof traffic of the vehicles 414A-B based on the traffic information, andcompare the change in behavior of the vehicles 414A-B to a predeterminedor typical pattern of traffic changes associated with approaching agiven construction zone.

In yet still another example, the information relating to detection ofthe construction zone may include map information related to prior orpreexisting maps. For example, the map information may includeinformation associated with traffic signs 416A-B, a number of lanes onthe road 404, locations of lane boundaries, etc. The prior maps may bepopulated with existing signs manually or through electronic detectionof the existing signs. However, the map information may not includeinformation relating to recent road changes due to temporary road workthat may cause changes to road lanes. For example, the map informationmay not include respective information relating to temporaryconstruction zone signs such as the construction zone signs 412A-B.

Additionally, each source of information of the plurality of sources ofinformation may be assigned a respective reliability metric. Thereliability metric may be indicative of a level of confidence of thedetection of the construction zone based on respective informationreceived from that source of information. As an example forillustration, the traffic information may be more reliable in detectingthe construction zone than the RADAR-based information; in other words,the computing device may be configured to detect, based on the trafficinformation, existence of a construction zone with level of confidencethat is higher than a respective level of confidence of detecting theconstruction zone based on the RADAR-based information. In this example,the source of the traffic information may be assigned a higherreliability metric than the RADAR unit, which may be the source of theRADAR-based information. In examples, the reliability metric may bedetermined based on previously collected data from a plurality ofdriving situations.

Referring back to FIG. 3, at block 304, the method 300 includesdetermining, using the computing device, a likelihood of existence ofthe construction zone on the road, based on the information andrespective reliability metrics of the plurality of sources ofinformation. As an example, in FIG. 4, based on the information relatingto detection of the construction zone and received from the plurality ofsources at the computing device configured to control the vehicle 402,the computing device may be configured to determine a likelihood ofexistence of the construction zone on the road 404.

In an example, the computing device may be configured to determine, fromthe image-based information received from the image-capture device, achange in road geometry due to the construction zone, and may assign thelikelihood based on the determined change. For example, the computingdevice may be configured to compare the determined change to a typicalchange associated with a typical construction zone, and determine thelikelihood based on the comparison. As another example, the computingdevice may be configured to identify, using image recognition techniquesknown in the art, construction zone objects (e.g., the construction zonecone(s) 406, the construction zone barrel(s) 408, the construction zonesigns 412A-B, the construction equipment 410A-B, or any otherconstruction zone indicators) depicted in images captured by theimage-capture device. In one example, the computing device may beconfigured to assign a respective likelihood of identification that maybe indicative of a level of confidence associated with identifyingconstruction zone objects, and determine the likelihood of the existenceof the construction zone based on the respective likelihoods ofidentification for the construction zone objects.

Similarly, the computing device may be configured to identify theconstruction zone objects based on the LIDAR-based and/or RADAR-basedinformation. As an example, the computing device may be configured toidentify a candidate construction zone object (a candidate constructionzone cone, barrel, or sign) represented by a set of points of a 3D pointcloud provided by the LIDAR sensor; and the computing device may beconfigured to assign a respective likelihood of identification for thecandidate construction zone object based on a respective level ofconfidence for identification of the object.

In an example, the computing device may be configured to compare a shapeof the candidate construction zone object (identified in the image-basedinformation, LIDAR-based information, or RADAR-based information) to oneor more predetermined shapes of typical construction zone objects; andalso may be configured to determine a match metric indicative of howsimilar the candidate construction zone object is to a givenpredetermined shape (e.g., a percentage of match between dimensionalcharacteristics of the shape of the candidate object and the givenpredetermined shape). The computing device may be configured todetermine the likelihood based on the match metric. The computing devicealso may be configured to use any of the techniques described below withregard to FIG. 9 to detect and/or identify construction zone objects.

In an example, the computing device may be configured to receive the mapinformation associated with the road 404, and the map information mayinclude locations and types of existing signs (e.g., the signs 416A-B)on the road 404. The computing device further may be configured todetermine a presence of a candidate construction zone sign (e.g., one orboth of the construction zone signs 412A-B), which may be missing fromthe map information. In one example, the candidate construction zonesign being missing from the map information may be indicative oftemporariness of the candidate construction zone sign, and thus mayindicate that the candidate construction zone sign is likely aconstruction zone sign. Accordingly, the computing device may assign arespective likelihood that the candidate construction zone sign isassociated with a construction zone. In an example, the computing devicemay be configured to update the map information to include respectivesign information associated with the candidate construction zone signand the likelihood of the existence of the construction zone on theroad, so as to allow other vehicles or drivers on the road 404 to becautious that there is a given likelihood of existence of a givenconstruction zone on the road 404. The computing device also may beconfigured to use any of the techniques described below with regard toFIGS. 5 and 7 to detect and/or identify construction zone signs.

In still another example, the computing device may be configured toreceive the traffic information indicative of behavior of the othervehicles 414A-B on the road 404. In this example, to determine thelikelihood, the computing device may be configured to determine a changein nominal speed and flow of traffic of the other vehicles 414A-B basedon the traffic information. The computing device may be configured tocompare the change in behavior of the other vehicles 414A-B with apredetermined or typical pattern of traffic changes associated withapproaching a given construction zone, and the computing device may beconfigured to determine the likelihood based on the comparison. In anexample, the computing device, in determining the likelihood based onthe comparison, may be configured to distinguish a given change intraffic associated with an accident site from a respective change intraffic associated with approaching a respective construction zone. Forexample, an accident site may be characterized by a congestion pointtowards which vehicles may slow down and accelerate once the congestionpoint is passed; alternatively, construction zones may be characterizedby a longer road section of changed speed and flow of traffic. Inanother example, the computing device may be configured to distinguishan accident site from a construction zone based on accident informationreceived from an accident broadcasting service.

In one example, the computing device may be configured to assign ordetermine a respective likelihood of existence of the construction zonefor each type or source of information (e.g., the image-basedinformation, LIDAR-based information, RADAR-based information, the mapinformation, and the traffic information) and further may be configuredto determine a single likelihood based on a combination of therespective likelihoods (e.g., a weighted combination of the respectivelikelihoods). For instance, the respective likelihood assigned to eachsource of information of the plurality of sources of information may bebased on the reliability metric assigned to that source of information.Also, in an example, based on a respective likelihood determined for asource of the plurality of sources of information, the computing devicemay be configured to enable a sensor or module coupled to the vehicle402 to receive information from another source of information to confirmexistence of the construction zone.

In another example, the computing device may be configured to generate aprobabilistic model (e.g., a Gaussian distribution), based on theinformation relating to detection of the construction zone received fromthe plurality of sources and the respective reliability metrics assignedto the plurality of sources, to determine the likelihood of theexistence of the construction zone. For example, the likelihood of theexistence of the construction zone may be determined as a function of aset of parameter values that are determined based on the informationfrom the plurality of sources and the respective reliability metrics. Inthis example, the likelihood may be defined as equal to probability ofan observed outcome (the existence of the construction zone) given thoseparameter values. Those skilled in the art will appreciate thatdetermining the likelihood function may involve distinguishing betweendiscrete probability distribution, continuous probability distribution,and mixed continuous-discrete distributions, and that several types oflikelihood exist such as log likelihood, relative likelihood,conditional likelihood, marginal likelihood, profile likelihood, andpartial likelihood.

In still another example, the computing device may be configured toprocess the information from the plurality of sources and the respectivereliability metrics through a classifier to determine the likelihood.The classifier can be defined as an algorithm or mathematical functionimplemented by a classification algorithm that maps input information(e.g., the information relating to detection of the construction zoneand the respective reliability metrics) to a class (e.g., existence ofthe construction zone).

Classification may involve identifying to which of a set of classes(e.g., existence or nonexistence of the construction zone) a newobservation may belong, on the basis of a training set of datacontaining observations (or instances) with a known class. Theindividual observations may be analyzed into a set of quantifiableproperties, known as various explanatory variables or features. As anexample, classification may include assigning a respective likelihood to“existence of construction zone” or “nonexistence of construction zone”classes as indicated by received information relating to detection ofthe construction zone (e.g., image-based information, LIDAR-basedinformation, RADAR-based information, map information, trafficinformation, etc.).

In one example, the classification may include a probabilisticclassification. Probabilistic classification algorithms may output aprobability of an instance (e.g., a driving situation or a group ofobservations indicated by the received information relating to thedetection of the construction zone) being a member of each of thepossible classes: “existence of construction zone” or “nonexistence ofconstruction zone”. Determining likelihood of the existence of theconstruction zone may be based on probability assigned to each class.Also, the probabilistic classification can output a confidence valueassociated with the existence of the construction zone.

Example classification algorithms may include Linear classifiers (e.g.,Fisher's linear discriminant, logistic regression, naive Bayes, andperceptron), Support vector machines (e.g., least squares support vectormachines), quadratic classifiers, kernel estimation (e.g., k-nearestneighbor), boosting, decision trees (e.g., random forests), neuralnetworks, Gene Expression Programming, Bayesian networks, hidden Markovmodels, and learning vector quantization. Other example classifiers arealso possible.

As an example for illustration, a linear classifier may be expressed asa linear function that assigns a score or likelihood to each possibleclass k (e.g., “existence of construction zone” or “nonexistence ofconstruction zone”) by combining a feature vector (vector of parametersassociated with the information relating to the detection of theconstruction zone and received from the plurality of sources and therespective reliability metrics) of an instance (e.g., a drivingsituation) with a vector of weights, using a dot product. Class with thehigher score or likelihood may be selected as a predicted class. Thistype of score function is known as a linear predictor function and mayhave this general form:

Score(X _(i) ,k)=β_(k) ·X _(i)  Equation (1)

where X_(i) is the feature vector for instance i, β_(k) is a vector ofweights corresponding to category k, and score(X_(i),k) is the scoreassociated with assigning instance i to category k.

As an example, a training computing device may be configured to receivetraining data for a plurality of driving situations of a given vehicle.For example, for each of the plurality of driving situations, respectivetraining data may include respective image-based information, respectiveLIDAR-based information, respective RADAR-based information, respectivetraffic information, and respective map information. Also, the trainingcomputing device may be configured to receive positive or negativeindication of existence of a respective construction zone correspondingto the respective training data for each of the driving situations.Further the training computing device may be configured to correlate,for each driving situation, the positive or negative indication with therespective training data; and determine parameters (e.g., vector ofweights for equation 1) of the classifier based on the correlations forthe plurality of driving situations. Further, in an example, thetraining computing device may be configured to determine a respectivereliability metric for each source of information based on thecorrelation. The parameters and respective reliability metrics of theplurality of sources of information may be provided to the computingdevice configured to control the vehicle 402 such that as the computingdevice receives the information, from the plurality of sources ofinformation, relating to the detection of the construction zone, thecomputing device may be configured to process the information throughthe classifier using the determined parameters of the classifier todetermine the likelihood.

In one example, the likelihood may be qualitative such as “low,”“medium,” or “high” or may be numerical such as a number on a scale, forexample. Other examples are possible.

Referring back to FIG. 3, at block 306, the method 300 includesmodifying, using the computing device, a control strategy associatedwith a driving behavior of the vehicle, based on the likelihood.

The control system of the vehicle may support multiple controlstrategies and associated driving behaviors that may be predetermined oradaptive to changes in a driving environment of the vehicle. Generally,a control strategy may comprise sets of rules associated with trafficinteraction in various driving contexts such as approaching aconstruction zone. The control strategy may comprise rules thatdetermine a speed of the vehicle and a lane that the vehicle may travelon while taking into account safety and traffic rules and concerns(e.g., changes in road geometry due to existence of a construction zone,vehicles stopped at an intersection and windows-of-opportunity in yieldsituation, lane tracking, speed control, distance from other vehicles onthe road, passing other vehicles, and queuing in stop-and-go traffic,and avoiding areas that may result in unsafe behavior such asoncoming-traffic lanes, etc.). For instance, in approaching aconstruction zone, the computing device may be configured to modify orselect, based on the determined likelihood of the existence of theconstruction zone, a control strategy comprising rules for actions thatcontrol the vehicle speed to safely maintain a distance with otherobjects and select a lane that is considered safest given road changesdue to the existence of the construction zone.

As an example, in FIG. 4, if the likelihood of the existence of theconstruction zone is high (e.g., exceeds a predetermined threshold), thecomputing device may be configured to utilize sensor information,received from on-board sensors on the vehicle 402 or off-board sensorsin communication with the computing device, in making a navigationdecision rather than preexisting map information that may not includeinformation and changes relating to the construction zone. Also, thecomputing device may be configured to utilize the sensor informationrather than the preexisting map information to estimate lane boundaries.For example, referring to FIG. 4, the computing device may be configuredto determine locations of construction zone markers (e.g., theconstruction zone cone(s) 406) rather than lane markers 418 on the road404 to estimate and follow the lane boundaries. As another example, thecomputing device may be configured to activate one or more sensors fordetection of construction workers 420 and making the navigation decisionbased on the detection.

In an example, a first control strategy may comprise a default drivingbehavior and a second control strategy may comprise a defensive drivingbehavior. Characteristics of a the defensive driving behavior maycomprise, for example, following a vehicle of the vehicles 414A-B,maintaining a predetermined safe distance with the vehicles 414A-B thatmay be larger than a distance maintained in the default drivingbehavior, turning-on lights, reducing a speed of the vehicle 402, andstopping the vehicle 402. In this example, the computing device of thevehicle 402 may be configured to compare the determined likelihood to athreshold likelihood, and the computing device may be configured toselect the first or the second control strategy, based on thecomparison. For example, if the determined likelihood is greater thanthe threshold likelihood, the computing device may be configured toselect the second driving behavior (e.g., the defensive drivingbehavior). If the determined likelihood is less than the thresholdlikelihood, the computing device may be configured to modify the controlstrategy to the first control strategy (e.g., select the default drivingbehavior).

In yet another example, alternatively or in addition to transitionbetween discrete control strategies (e.g., the first control strategyand the second control strategy) the computing device may be configuredto select from a continuum of driving modes or states based on thedetermined likelihood. In still another example, the computing devicemay be configured to select a discrete control strategy and also may beconfigured to select a driving mode from a continuum of driving modeswithin the selected discrete control strategy. In this example, a givencontrol strategy may comprise multiple sets of driving rules, where aset of driving rules describe actions for control of speed and directionof the vehicle 402. The computing device further may be configured tocause a smooth transition from a given set of driving rules to anotherset of driving rules of the multiple sets of driving rules, based on thedetermined likelihood. A smooth transition may indicate that thetransition from the given set of rules to another may not be perceivedby a passenger in the vehicle 402 as a sudden or jerky change in a speedor direction of the vehicle 402, for example.

In an example, a given control strategy may comprise a program orcomputer instructions that characterize actuators controlling thevehicle 402 (e.g., throttle, steering gear, brake, accelerator, ortransmission shifter) based on the determined likelihood. The givencontrol strategy may include action sets ranked by priority, and theaction sets may include alternative actions that the vehicle 402 maytake to accomplish a task (e.g., driving from one location to another).The alternative actions may be ranked based on the determinedlikelihood, for example. Also, the computing device may be configured toselect an action to be performed and, optionally, modified based on thedetermined likelihood.

In another example, multiple control strategies (e.g., programs) maycontinuously propose actions to the computing device. The computingdevice may be configured to decide which strategy may be selected or maybe configured to modify the control strategy based on a weighted set ofgoals (safety, speed, etc.), for example. Weights of the weighted set ofgoals may be a function of the determined likelihood. Based on anevaluation of the weighted set of goals, the computing device, forexample, may be configured to rank the multiple control strategies andrespective action sets and select or modify a given strategy and arespective action set based on the ranking.

These examples and driving situations are for illustration only. Otherexamples and control strategies and driving behaviors are possible aswell.

Referring back to FIG. 3, at block 308, the method 300 includescontrolling, using the computing device, the vehicle based on themodified control strategy. In an example, the computing device may beconfigured to control actuators of the vehicle using an action set orrule set associated with the modified control strategy. For instance,the computing device may be configured to adjust translational velocity,or rotational velocity, or both, of the vehicle based on the modifieddriving behavior.

As an example, in FIG. 4, controlling the vehicle 402 may comprisedetermining a desired path of the vehicle, based on the likelihood. Inone example, the computing device may have determined a high likelihoodthat a construction zone exists on the road 404 on which the vehicle 402is travelling. In this example, the computing device may be configuredto take into account lane boundary indicated by the lane markers 418 onthe road 404 as a soft constraint (i.e., the lane boundary can beviolated if a safer path is determined) when determining the desiredpath. The computing device thus may be configured to determine a numberand locations of the construction zone cone(s) 406 that may form amodified lane boundary; and may be configured to adhere to the modifiedlane boundary instead of the lane boundary indicated by the lane markers418.

As shown in FIG. 4, the vehicle 402 may be approaching the constructionzone on the road 404, and the computing device may be configured tocontrol the vehicle 402 according to a defensive driving behavior tosafely navigate the construction zone. For example, the computing devicemay be configured to reduce speed of the vehicle 402, cause the vehicle402 to change lanes and adhere to the modified lane boundary formed bythe construction zone cone(s) 406, shift to a position behind thevehicle 414A, and follow the vehicle 414A while keeping a predeterminedsafe distance.

In one example, in addition to determining the likelihood of theexistence of the construction zone, the computing device may beconfigured to determine or estimate a severity of changes to the road404 due to the existence of the construction zone. The computing devicemay be configured to modify the control strategy further based on theseverity of the changes. As an example, in FIG. 4, the computing devicemay be configured to determine, based on the construction equipment410A-B, number and locations of the construction zone cone(s) 406 andbarrel(s) 408, how severe the changes (e.g., lane closure, shifts, etc.)to the road 404 are, and control the vehicle 402 in accordance with thedefensive driving behavior. In another example, the construction zonemay comprise less severe changes. For example, the construction zone maycomprise a worker that may be painting on a curb lane on a side of theroad 404. In this example, changes to the road 404 may be less severethan changes depicted in FIG. 4, and the computing device may beconfigured to reduce the speed of the vehicle 402 as opposed to stop thevehicle 402 or cause the vehicle 402 to change lanes, for example.

These control actions and driving situations are for illustration only.Other actions and situations are possible as well. In one example, thecomputing device may be configured to control the vehicle based on themodified control strategy as an interim control until a human driver cantake control of the vehicle.

As described above with respect to FIGS. 3 and 4, the computing devicemay be configured to determine the likelihood of the existence of theconstruction zone based on identification or detection of a constructionzone sign (e.g., the construction zone sign 412A) that may be indicativeof the construction zone.

FIG. 5 is a flow chart of a method 500 for detection of a constructionzone sign, in accordance with an example embodiment. FIGS. 6A-6Billustrate images of a road and vicinity of the road the vehicle istravelling on, in accordance with an example embodiment, and FIGS. 6C-6Dillustrate portions of the images of the road and the vicinity of theroad depicting sides of the road at a predetermined height range, inaccordance with an example embodiment. FIGS. 5 and 6A-6D will bedescribed together.

The method 500 may include one or more operations, functions, or actionsas illustrated by one or more of blocks 502-512. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

At block 502, the method 500 includes receiving, at a computing deviceconfigured to control a vehicle, from an image-capture device coupled tothe computing device, one or more images of a vicinity of a road onwhich the vehicle is travelling. The computing device may be onboard thevehicle or may be off-board but in wireless communication with thevehicle, for example. Also, the computing device may be configured tocontrol the vehicle in an autonomous or semi-autonomous operation mode.Further, an image-capture device (e.g., the camera 134 in FIG. 1 or thecamera 210 in FIG. 2) may be coupled to the vehicle and in communicationwith the computing device. The image-capture device may be configured tocapture images or video of the road and vicinity of the road on whichthe vehicle is travelling on.

FIGS. 6A-6B, for example, illustrate example images 602 and 604,respectively, captured by the image-capture device coupled to thevehicle 402 in FIG. 4. In an example, the image-capture device may beconfigured to continuously capture still images or a video from whichthe still images can be extracted. In one example, one or moreimage-capture devices may be coupled to the vehicle 402; the one or moreimage-capture devices may be configured to capture the images frommultiple views to take into account surroundings of the vehicle 402 androad condition from all directions.

Referring back to FIG. 5, at block 504, the method 500 includesdetermining, using the computing device, one or more image portions inthe one or more images, and the one or more image portions may depictsides of the road at a predetermined height range. In some examples, thepredetermined height range may correspond to a height range that istypically used for construction zone signs. In many jurisdictions,construction zones on roads are regulated by standard specifications andrules, which may be used to define the predetermined height range. Anexample rule may state that a construction zone sign indicatingexistence of a construction zone on the road may be placed at a givenlocation continuously for longer than three days and may be mounted on apost on a side of the road. Further, another rule may specify that aminimum sign mounting height for a temporary warning construction zonesign, for example, may be 1 foot above road ground level. In otherexamples, in addition to or alternative to the minimum sign mountingheight, a height range can be specified, i.e., a height range for atemporary warning construction zone sign may be between 1 foot and 6feet, for example. In some locations where the construction zone signmay be located behind a traffic control device such as a traffic safetydrum or temporary barrier, the minimum height may be raised to 5 feet inorder to provide additional visibility. Additionally or alternatively, aheight range can be specified to be between 5 feet and 11 feet, forexample. These numbers and rules are for illustration only. Otherstandards and rules are possible. In some examples, the predeterminedheight range or minimum height of a typical construction zone sign maybe dependent on location (e.g., geographic region, which state in theUnited States of America, country, etc.).

The computing device may be configured to determine portions, in theimages captured by the image-capture device, which may depict road sidesat the predetermined height range of a typical construction zone signaccording to the standard specifications. As an example, in FIG. 6A, thecomputing device may be configured to determine a portion 606 in theimage 602 depicting a side of the road 404 at a predetermined heightrange specified for typical construction zone signs according to thestandard specifications. Similarly, in FIG. 6B, the computing device maybe configured to determine a portion 608 in the image 604 depictinganother side of the road 404 at the predetermined height range. FIG. 6Cillustrates the image portion 606 of the image 602 illustrated in FIG.6A, and FIG. 6D illustrates the image portion 608 of the image 604illustrated in FIG. 6B.

Referring back to FIG. 5, at block 506, the method 500 includesdetecting, using the computing device, a construction zone sign in theone or more image portions. The standard specifications also may includerules for shape, color, pattern, and retroreflective characteristics oftypical construction zone signs. As an example for illustration, thestandard specifications may specify that a typical construction zonesign may be a 48 inches×48 inches diamond shape with black letters ofsymbols on an orange background having a standard type of reflectivesheeting. These specifications are for illustration only, and otherspecifications are possible.

As an example, referring to FIGS. 6A-6D, the computing device may beconfigured to detect candidate construction zone signs, such as the sign412A and the sign 416B in the image portions 608 and 606, respectively.The computing device further may be configured to determine, using imagerecognition techniques known in the art for example, whether a candidateconstruction zone sign relates to a construction zone, based on one ormore of the shape, color, pattern, and reflective characteristics of thecandidate construction zone sign as compared to the standardspecifications of typical construction zone signs. For example, thecomputing device may be configured, based on the comparison, todetermine that the sign 412A is a construction zone sign, while the sign416B is not.

In an example to illustrate use of image recognition, the computingdevice may be configured to compare an object detected, in the one ormore image portions, to a template of the typical construction zonesign. For example, the computing device may be configured to identifyfeatures of the object such as color, shape, edges, and corners of theobject in the one or more image portions. Then, the computing device maybe configured to compare these features to orange/yellow color, diamondshape with sharp edges, and corners (i.e., “corner signature”) of thetypical construction zone sign. The computing device may be configuredto process the features (e.g., color, shape, etc.) or parametersrepresentative of the features of the object through a classifier todetermine whether the features of the object match typical features ofthe typical construction zone sign. The classifier can map inputinformation (e.g., the features of the object) to a class (e.g., theobject represents a construction zone sign). Examples of classifiers,training data, and classification algorithms are described above withregard to block 304 of the method 300 illustrated in FIG. 3.

In an example, the computing device may be configured to use informationreceived from other sensors or units coupled to the vehicle 402, inaddition to image-based information received from the image-capturedevice, to confirm or validate detection of a construction zone sign.For example, the computing device may be configured to assign ordetermine, based on the image-based information, a first likelihood thata candidate construction zone sign in the image portions relates to aconstruction zone. Further, the computing device may be configured toreceive, from a LIDAR sensor (e.g., the LIDAR unit 132 in FIG. 1)coupled to the vehicle 402 and in communication with the computingdevice, LIDAR-based information that includes a 3D point cloudcorresponding to the image portions (e.g., the image portion 608)depicting the candidate construction zone sign (e.g., the sign 412A).The 3D point cloud may comprise a set of points based on light emittedfrom the LIDAR and reflected from a surface of the candidateconstruction zone sign. The computing device may be configured todetermine a second likelihood that the candidate construction zone signrelates to the construction zone, based on the LIDAR-based information,and confirm existence or detection of the construction zone sign basedon the first likelihood and the second likelihood.

In another example, in addition to or alternative to receiving theLIDAR-based information, the computing device may be configured toreceive, from a RADAR sensor (e.g., the RADAR unit 130 in FIG. 1)coupled to the computing device, RADAR-based information relating tolocation and characteristics of the candidate construction zone sign.The RADAR sensor may be configured to emit radio waves and receive backthe emitted radio waves that bounced off the surface of the candidateconstruction zone sign. The received signals or RADAR-based informationmay be indicative, for example, of dimensional characteristics of thecandidate construction zone sign, and may indicate that the candidateconstruction zone sign is stationary. The computing device may beconfigured to determine a third likelihood that the candidateconstruction zone sign relates to the construction zone, based on theRADAR-based information, e.g., based on a comparison of thecharacteristics of the candidate construction zone sign to standardcharacteristics of a typical construction zone sign. Further, thecomputing device may be configured to detect the construction zone signbased on the first likelihood, the second likelihood, and the thirdlikelihood.

As an example, the computing device may be configured to determine anoverall likelihood that is a function of the first likelihood, thesecond likelihood, and the third likelihood (e.g., a weightedcombination of the first likelihood, the second likelihood, and thethird likelihood), and the computing device may be configured to detectthe construction zone sign based on the overall likelihood.

In one example, the computing device may be configured to detect theconstruction zone sign based on information received from multiplesources such as the image-capture device, the LIDAR sensor, and theRADAR sensor; but, in another example, the computing device may beconfigured to detect the construction zone sign based on a subset ofinformation received from a subset of the multiple sources. For example,images captured by the image-capture device may be blurred due to amalfunction of the image-capture device. As another example, details ofthe road 404 may be obscured in the images because of fog. In theseexamples, the computing device may be configured to detect theconstruction zone sign based on information received from the LIDARand/or RADAR units and may be configured to disregard the informationreceived from the image-capture device.

In another example, the vehicle 402 may be travelling in a portion ofthe road 404 where some electric noise or jamming signals may exist, andthus the LIDAR and/or RADAR signals may not operate correctly. In thiscase, the computing device may be configured to detect the constructionzone sign based on information received from the image-capture device,and may be configured to disregard the information received from theLIDAR and/or RADAR units.

In one example, the computing device may be configured to rank theplurality of sources of information based on a condition of the road 404(e.g., fog, electronic jamming, etc.) and/or based on the respectivereliability metric assigned to each source of the plurality of sources.The ranking may be indicative of which sensor(s) to rely on or give moreweight to in detecting the construction zone sign. As an example, if fogis present in a portion of the road, then the LIDAR and RADAR sensorsmay be ranked higher than the image-based device, and informationreceived from the LIDAR and/or RADAR sensor may be given more weightthan respective information received from the image-capture device.

Referring back to FIG. 5, at block 508, the method 500 includesdetermining, using the computing device, a type of the construction zonesign in the one or more image portions. Various types of constructionzone signs may exist. One construction zone sign type may be related toregulating speed limits when approaching and passing through aconstruction zone on a road. Another construction zone sign type may berelated to lane changes, closure, reduction, merger, etc. Still anotherconstruction zone sign type may be related to temporary changes todirection of travel on the road. Example types of construction zonesigns may include: “Right Lane Closed Ahead,” “Road Work Ahead,” “BePrepared to Stop,” “Road Construction 1500 ft,” “One Lane Road Ahead,”“Reduced Speed Limit 30,” “Shoulder Work,” etc. Other example types arepossible.

In an example, the computing device of the vehicle may be configured todetermine a type of the detected construction zone based on shape,color, typeface of words, etc. of the construction zone sign. As anexample, the computing device may be configured to use image recognitiontechniques to identify the type (e.g., shape of, or words written on,the construction zone sign) from an image of the detected constructionzone sign.

As described above with respect to block 506, the computing device maybe configured to utilize image recognition techniques to compare anobject to a template of a typical construction zone sign to detect aconstruction zone sign. In an example, to determine the type of thedetected construction zone sign, the computing device may be configuredto compare portions of the detected construction zone sign tosub-templates of typical construction zone signs. In one example, thecomputing device may be configured to identify individual words orcharacters typed on the detected construction zone sign, and compare theidentified words or characters to corresponding sub-templates of typicalconstruction zone signs. In another example, the computing device may beconfigured to determine spacing between the characters or the words,and/or spacing between the words and edges of the detected constructionzone sign. In still another example, the computing device may beconfigured to identify a font in which the words or characters areprinted on the detected construction zone sign and compare theidentified font to fonts sub-template associated with typicalconstruction zone signs.

As an example, the computing device may be configured to detect aconstruction zone sign having the words “Road Work Ahead” typed on thedetected construction zone sign. The computing device may be configuredto extract individual characters or words “Road,” “Work,” “Ahead,” inthe one or more image portions, and compare these words andcharacteristics of these words (e.g., fonts, letter sizes, etc.) tocorresponding sub-templates typical construction zone signs. Also, thecomputing device may be configured to compare spacing between the threewords and spacing between letters forming the words to correspondingsub-templates. Further, the computing device may be configured tocompare spacing between the word “Road” and a left edge of the detectedconstruction zone sign and the spacing between the word “Ahead” and aright edge of the detected construction zone sign to correspondingsub-templates. Based on these comparisons, the computing device may beconfigured to determine the type of the detected construction zone sign.

These features (e.g., characters, words, fonts, spacing, etc.) areexamples for illustrations, and other features can be used and comparedto typical features (i.e., sub-templates) of typical construction zonesigns to determine the type of the detected construction zone sign.

In one example, in addition to or alternative to using imagerecognition, based on the RADAR-based information, the computing devicemay be configured to determine shape and dimensions of the constructionzone sign and infer the type and associated road changes from thedetermined shape and dimensions. Other examples are possible.

At block 510, the method 500 includes modifying, using the computingdevice, a control strategy associated with a driving behavior of thevehicle, based on the type of the construction zone sign. The roadchanges due to existence of a construction zone on the road may beindicated by the type of the construction zone sign existing on ahead ofthe construction zone. The computing device may be configured to modifycontrol strategy of the vehicle based on the determined type of theconstruction zone sign.

Examples of modifying the control strategy are described above withregard to block 306 of the method 300 illustrated in FIG. 3. Asexamples, the computing device may be configured to determine whether alane shift and/or speed change are required as indicated by the type;utilize sensor information received from on-board or off-board sensorsin making a navigation decision rather than preexisting map information;utilize the sensor information to estimate lane boundaries rather thanthe preexisting map information; determine locations of constructionzone cones or barrels rather than lane markers on the road to estimateand follow the lane boundaries; and activate one or more sensors fordetection of construction workers and making the navigation decisionbased on the detection. These examples and driving situations are forillustration only. Other examples and control strategies and drivingbehaviors are possible as well.

At block 512, the method 500 includes controlling, using the computingdevice, the vehicle based on the modified control strategy. Examples ofcontrolling the vehicle based on the modified control strategy aredescribed above with regard to block 308 of the method 300 illustratedin FIG. 3. As examples, the computing device may be configured to adjusttranslational velocity, or rotational velocity, or both, of the vehiclebased on the modified driving behavior in order to follow anothervehicle; maintain a predetermined safe distance with other vehicles;turn-on lights; reduce a speed of the vehicle; shift lanes; and stop thevehicle. These control actions and driving situations are forillustration only. Other actions and situations are possible as well.

As described with respect to block 506 of the method 500 illustrated inFIG. 5, the computing device may be configured to detect or confirmdetection of the construction zone sign based on information receivedfrom a LIDAR sensor coupled to the vehicle and in communication with thecomputing device.

FIG. 7 is a flow chart of a method 700 for detection of the constructionzone sign using LIDAR-based information, in accordance with an exampleembodiment. FIG. 8A illustrates LIDAR-based detection of theconstruction zone sign at a height greater than a threshold height froma surface of the road, in accordance with an example embodiment. FIG. 8Billustrates a LIDAR-based image depicting the area at the height greaterthan the threshold height from the surface of the road, in accordancewith an example embodiment. FIGS. 7 and 8A-8B will be describedtogether.

The method 700 may include one or more operations, functions, or actionsas illustrated by one or more of blocks 702-712. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

At block 702, the method 700 includes receiving, at a computing deviceconfigured to control a vehicle, from a light detection and ranging(LIDAR) sensor coupled to the computing device, LIDAR-based informationcomprising (i) a three-dimensional (3D) point cloud of a vicinity of aroad on which the vehicle is travelling, where the 3D point cloud maycomprise points corresponding to light emitted from the LIDAR andreflected from one or more objects in the vicinity of the road, and (ii)intensity values of the reflected light for the points. A LIDAR sensoror unit (e.g., the LIDAR unit 132 in FIG. 1) may be coupled to thevehicle and in communication with the computing device configured tocontrol the vehicle. As described with respect to the LIDAR unit 132 inFIG. 1, LIDAR operation may involve an optical remote sensing technologythat enables measuring properties of scattered light to find rangeand/or other information of a distant target. The LIDAR sensor/unit, forexample, may be configured to emit laser pulses as a beam, and scan thebeam to generate two dimensional or three dimensional range matrices. Inan example, the range matrices may be used to determine distance to anobject or surface by measuring time delay between transmission of apulse and detection of a respective reflected signal.

In another example, the LIDAR sensor may be configured to rapidly scanan environment surrounding the vehicle in three dimensions. In someexamples, more than one LIDAR sensor may be coupled to the vehicle toscan a complete 360° horizon of the vehicle. The LIDAR sensor may beconfigured to provide to the computing device a cloud of point datarepresenting objects, which have been hit by the laser, on the road andthe vicinity of the road. The points may be represented by the LIDARsensor in terms of azimuth and elevation angles, in addition to range,which can be converted to (X, Y, Z) point data relative to a localcoordinate frame attached to the vehicle. Additionally, the LIDAR sensormay be configured to provide to the computing device intensity values ofthe light or laser reflected off the objects.

At block 704, the method 700 includes determining, using the computingdevice, a set of points in the 3D point cloud representing an area at aheight greater than a threshold height from a surface of the road. Asdescribed with respect to the method 500, construction zones on roadsmay be regulated by standard specifications and rules. A minimum signmounting height may be specified for a typical construction zone sign,for example. FIG. 8A illustrates the vehicle 402 travelling on the road404 and approaching a construction zone indicated by the constructionzone sign 412A. The LIDAR sensor coupled to the vehicle 402 may bescanning the horizon and providing the computing device with a 3D pointcloud of the road 404 and a vicinity (e.g., sides) of the road 404.Further, the computing device may be configured to determine an area 802at a height greater than a threshold height 804; the threshold height804 may be the minimum sign mounting height specified for a typicalconstruction zone sign according to the standard specifications ofconstruction zones, for example. FIG. 8B illustrates a LIDAR-based image806 including a set of points (e.g., a subset of the 3D point cloud)representing or corresponding to the determined area 802.

Referring back to FIG. 7, at block 706, the method 700 includesestimating, using the computing device, a shape associated with the setof points. The computing device may be configured to identify orestimate a shape depicted by the set of points representing the area atthe height greater than the threshold height. For example, the computingdevice may be configured to estimate dimensional characteristics of theshape. In an example, the computing device may be configured to fit apredetermined shape to the shape depicted in the set of point toestimate the shape. As an example, in FIG. 8B, the computing device maybe configured to estimate a diamond shape 808 in the set of pointincluded in the LIDAR-based image 806.

Referring back to FIG. 7, at block 708, the method 700 includesdetermining, using the computing device, a likelihood that the set ofpoints depicts a construction zone sign, based on the estimated shapeand respective intensity values relating to the set of points. In anexample, referring to FIG. 8B, the computing device may be configured tomatch or compare the estimated shape 808 to one or more shapes oftypical construction zone signs; and the computing device may beconfigured to determine a match metric indicative of how similar theestimated shape 808 is to a given predetermined shape (e.g., apercentage of match between dimensional characteristics of the estimatedshape 808 and a diamond shape of a typical construction zone sign). Inone example, the computing device may be configured to identify edges ofthe estimated shape 808 and match a shape formed by the edges to atypical diamond shape of typical construction zone signs. The likelihoodmay be determined based on the match metric, for example.

Further, typical construction zone signs may be required by the standardspecifications of construction zones to be made of a retroreflectivesheeting materials such as glass beads or prisms, and the computingdevice may be configured to compare intensity values of points formingthe estimated shaped 808 to a threshold intensity value of theretroreflective sheeting material. Based on the comparison, thecomputing device may be configured to confirm that the estimated shape808 may represent a given construction zone sign. For example, if theintensity values are close to or within a predetermined value of thethreshold intensity value, the computing device may be configured todetermine a high likelihood that that the estimated shape 808 mayrepresent a construction zone sign.

In an example, the computing device may be configured to determine afirst likelihood based on a comparison of the estimated shape 808 to apredetermined shape of a typical construction zone sign, and may beconfigured to determine a second likelihood based on a comparison of theintensity values to the threshold intensity value. The computing devicemay be configured to combine the first likelihood and the secondlikelihood to determine a single likelihood that the set of points,which includes the points forming the estimated shape 808, depicts aconstruction zone sign.

In another example, the computing device may be configured to generate aprobabilistic model (e.g., a Gaussian distribution), based on theestimated shape 808 (e.g., dimensional characteristics of the estimateshape 808) and the intensity values, to determine the likelihood thatthe set of points depicts a construction zone sign. For example, thelikelihood may be determined as a function of a set of parameter valuesthat are determined based on dimensions of the estimated shape 808 andthe respective intensity values. In this example, the likelihood may bedefined as equal to probability of an observed outcome (the estimatedshape 808 represents a construction zone sign) given those parametervalues.

In still another example, the computing device may be configured tocluster points (e.g., the points forming the estimated shape 808)depicted in the LIDAR-based image 806 together into a cluster, based onlocations of the points or relative locations of the points to eachother. The computing device further may be configured to extract fromthe cluster of points a set of features (e.g., dimensionalcharacteristics of the estimated shape 808, and the intensity values ofthe points forming the estimated shape 808). The computing device may beconfigured to process this set of features through a classifier todetermine the likelihood. The classifier can map input information(e.g., the set of features extracted from the cluster of points) to aclass (e.g., the cluster represents a construction zone sign). Examplesof classifiers, training data, and classification algorithms aredescribed above with regard to block 304 of the method 300 illustratedin FIG. 3.

In one example, the likelihood may be qualitative such as “low,”“medium,” “high” or may be numerical such as a number on a scale, forexample. Other examples are possible.

Referring back to FIG. 7, at block 710, the method 700 includesmodifying, using the computing device, a control strategy associatedwith a driving behavior of the vehicle, based on the likelihood. Basedon the likelihood (e.g., the likelihood exceeds a predeterminedthreshold), the computing device may be configured to determineexistence of a construction zone sign indicative of an approachingconstruction zone. Further, the computing device may be configured todetermine a type of the construction zone sign to determine severity ofroad changes due to existence of the construction zone on the road. Forexample, the computing device may be configured to modify controlstrategy of the vehicle based on the determined type of the constructionzone sign.

As described above with respect to block 508 of the method 500 in FIG.5, various types of construction zone signs may exist to regulate speedlimits when approaching and passing through the construction zone,describe lane changes, closure, reduction, merger, etc., and describetemporary changes to direction of travel on the road, for example. Thecomputing device of the vehicle may be configured to determine the typeof the detected construction zone sign based on shape, color, typefaceof words, etc. of the detected construction zone sign.

Examples of modifying the control strategy are described above withregard to block 306 of the method 300 illustrated in FIG. 3.

At block 712, the method 700 includes controlling, using the computingdevice, the vehicle based on the modified control strategy. Controllingthe vehicle may include adjusting translational velocity, or rotationalvelocity, or both, of the vehicle based on the modified drivingbehavior. Examples of controlling the vehicle based on the modifiedcontrol strategy are described above with regard to block 308 of themethod 300 illustrated in FIG. 3.

In addition to or alternative to detection of the construction zone signusing the LIDAR-based information, the computing device may beconfigured to detect construction zone objects (e.g., cones, barrels,equipment, vests, chevrons, etc.) using the LIDAR-based information.

FIG. 9 is a flow chart of a method for detection of construction zoneobjects using LIDAR-based information, in accordance with an exampleembodiment. FIG. 10A illustrates LIDAR-based detection of constructionzone cones in an area within a threshold distance from a surface of theroad, in accordance with an example embodiment. FIG. 10B illustrates aLIDAR-based image depicting the area within the threshold distance fromthe surface of the road, in accordance with an example embodiment. FIG.10C illustrates LIDAR-based detection of construction zone cones forminga lane boundary, in accordance with an example embodiment. FIG. 10Dillustrates a LIDAR-based image depicting construction zone conesforming a lane boundary, in accordance with an example embodiment. FIGS.9 and 10A-10D will be described together. Detection of construction zonecones is used herein to illustrate the method 900; however, otherconstruction zone objects (e.g., construction zone barrels, equipment,vests, chevrons, etc.) can be detected using the method 900 as well.

The method 900 may include one or more operations, functions, or actionsas illustrated by one or more of blocks 902-914. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

At block 902, the method 900 includes receiving, at a computing deviceconfigured to control a vehicle, from a light detection and ranging(LIDAR) sensor coupled to the computing device, LIDAR-based informationrelating to a three-dimensional (3D) point cloud of a road on which thevehicle is travelling, where the 3D point cloud may comprise pointscorresponding to light emitted from the LIDAR and reflected from one ormore objects on the road. A LIDAR sensor or unit (e.g., the LIDAR unit132 in FIG. 1) may be coupled to the vehicle and in communication withthe computing device. As described above with respect to the LIDAR unit132 in FIG. 1, and block 702 of the method 700 illustrated in FIG. 7,the LIDAR sensor may be configured to provide to the computing device acloud of point data representing objects, on the road and the vicinityof the road. The points may be represented by the LIDAR sensor in termsof azimuth and elevation angles, in addition to range, which can beconverted to (X, Y, Z) point data relative to a local coordinate frameattached to the vehicle.

At block 904, the method 900 includes determining, using the computingdevice, one or more sets of points in the 3D point cloud representing anarea within a threshold distance from a surface of the road. Asdescribed above with respect to the methods 500 and 700, constructionzones on roads may be regulated by standard specifications and rules. Asan example, traffic safety cones may be used to separate and guidetraffic past a construction zone work area. Cones may be specified to beabout 18 inches tall, for example. In another example, for high speedand high volume of traffic, or nighttime operations, the cones may bespecified to be 28 inches tall, and retro-reflectorized, or comprisingbands made of retroreflective material. These examples are forillustration only, and other examples are possible.

FIG. 10A illustrates the vehicle 402 travelling on the road 404 andapproaching a construction zone indicated by the construction zone cone406. The LIDAR sensor coupled to the vehicle 402 may be configured toscan the horizon and provide the computing device with a 3D point cloudof the road 404 and a vicinity of the road 404. Further, the computingdevice may be configured to determine an area 1002 within a thresholddistance 1004 of a surface of the road 404. For example, the thresholddistance 1004 may be about 30 inches or more to include cones ofstandardized lengths (e.g., 18 inches or 28 inches). Other thresholddistances are possible based on the standard specifications regulating aparticular construction zone. FIG. 10B illustrates a LIDAR-based image1006 including sets of points representing objects in the area 1002.

Referring back to FIG. 9, at block 906, the method 900 includesidentifying one or more construction zone objects in the one or moresets of points. For example, the computing device may be configured toidentify shapes of objects represented by the sets of points of theLIDAR-based 3D point cloud. For example, the computing device may beconfigured to estimate characteristics (e.g., dimensionalcharacteristics) of a shape of an object depicted by a set of points,and may be configured to fit a predetermined shape to the shape toidentify the object. As an example, in FIG. 10B, the computing devicemay be configured to identify a construction zone cone 1008 in theLIDAR-based image 1006.

In an example, to identify the construction zone objects in the sets ofpoints, the computing device may be configured to determine, for eachidentified construction zone object, a respective likelihood of theidentification. As an example, in FIG. 10B, the computing device may beconfigured to determine a shape of the cone 1008 defined by respectivepoints of a set of points representing the cone 1008. Further, thecomputing device may be configured to match the shape to one or moreshapes of standard construction zone cones. The computing device may beconfigured to determine a match metric indicative of how similar theshape is to a given standard shape of a typical construction zone cone(e.g., a percentage of match between dimensional characteristics of theshape and the given standard shape). The respective likelihood may bedetermined based on the match metric.

In another example, in addition to or alternative to identifying thecone 1008 based on shape, the computing device may be configured tocluster points (e.g., the points forming the cone 1008) depicted in theLIDAR-based image 1006 together into a cluster, based on locations ofthe points or relative locations of the points to each other. Thecomputing device further may be configured to extract from the clusterof points a set of features (e.g., minimum height of the points, maximumheight of the points, number of the points, width of the cluster ofpoints, general statistics of the points at varying heights, etc.). Thecomputing device may be configured to process this set of featuresthrough a classifier to determine whether the cluster of pointsrepresent a given construction zone cone. The classifier can map inputinformation (e.g., the set of features extracted from the cluster ofpoints) to a class (e.g., the cluster represents a construction zonecone). Examples of classifiers, training data, and classificationalgorithms are described above with regard to block 304 of the method300 illustrated in FIG. 3.

Further, typical construction zone cones may be required by the standardspecifications of construction zones to be made of a retroreflectivesheeting materials such as glass beads or prisms, and the computingdevice may be configured to compare intensity values of points formingthe cone 1008 to a threshold intensity value of the retroreflectivesheeting material. Based on the comparison, the computing device may beconfigured to confirm identification of the cone 1008, for example.

In some examples, the computing device may be configured to excludecones that are away from the road by a certain distance, since suchcones may indicate a work zone that is away from the road and may notaffect traffic. Also, in an example, the computing device may beconfigured to exclude sets of points that represent objects that clearlycannot be construction zone cones based on size as compared to a typicalsize of typical construction zone cones (e.g., too large or too small tobe construction zone cones).

In an example, for reliable identification of a construction zone cone,the computing device may be configured to identify the construction zonecone based on LIDAR-based information received from two (or more)consecutive scans by the LIDAR to confirm the identification and filterout false identification caused by electronic or signal noise in asingle scan.

Referring back to FIG. 9, at block 908, the method 900 includesdetermining, using the computing device, a number and locations of theone or more construction zone objects. As an example, in addition tospecifying dimensional characteristics and reflective properties oftypical construction zone cones, the standard specifications forconstruction zones also may specify requirements for number of andspacing between the cones. In an example, tighter spacing may bespecified, under some conditions, to enhance guidance of vehicles anddrivers. Table 1 illustrates an example of minimum spacing betweenconstruction zone cones based on speed limits.

TABLE 1 Spacing for Speed A Spacing for Speed B Speed A: 50 mph 40 ft 80ft Speed B: 70 mph Speed A: 35 mph 30 ft 60 ft Speed B: 45 mph Speed A:20 mph 20 ft 40 ft Speed B: 30 mph

These examples are for illustration only. Other examples of spacingrequirements are possible as well.

In an example, if respective likelihoods for identification of theidentified cones exceed a threshold likelihood, the computing device maybe configured to further determine the number and locations of theidentified cones. In FIG. 10C, the computing device may be configured todetect or identify, based on the LIDAR-based information, the cone(s)406 and also determine number of the cone(s) 406 as well as locations orrelative locations of the cone(s) 406 with respect to each other. Forexample, the computing device may be configured to determine a distance1010 between the cone(s), and compare the distance 1010 with apredetermined distance (or spacing) specified in the standardspecifications.

FIG. 10D illustrates a LIDAR-based image 1011 including sets of pointsrepresenting construction zone cones 1012A-D. In addition to detectingor identifying the construction zone cones 1012A-D in the LIDAR-basedimage 1011, the computing device may be configured to estimate arespective distance between pairs of cones.

Referring back to FIG. 9, at block 910, the method 900 includesdetermining, using the computing device, a likelihood of existence of aconstruction zone, based on the number and locations of the one or moreconstruction zone objects. As an example, a single cone on a side of theroad may not be indicative of an active construction zone. Therefore, inaddition to detecting presence of or identifying cones on the road, thecomputing device may be configured, for example, to determine, based onthe number and locations (e.g., relative distance) of the cones, thatthe cones may form a lane boundary and are within a predetermineddistance of each other, which may be indicative of an activeconstruction zone causing road changes. The computing device thus may beconfigured to determine a likelihood or confirm that the cones areindicative of a construction zone, based on the determined number andlocations of the cones.

In an example, the computing device may be configured to determine thenumber and locations of the construction zone cones based on theLIDAR-based information, and compare a pattern formed by the identifiedconstruction zone cones to a typical pattern formed by construction zonecones in a typical construction zone (e.g., pattern of cones forming alane boundary). The computing device may be configured to determine thatthe detected construction zone cones are associated with a constructionzone based on the comparison, and determine the likelihood accordingly.

In another example, the computing device may be configured to generate aprobabilistic model (e.g., a Gaussian distribution), based on thedetermined number and locations of the cones to determine the likelihoodof existence of the construction zone. For example, the likelihood maybe determined as a function of a set of parameter values that aredetermined based on the number and locations of the identified cones. Inthis example, the likelihood may be defined as equal to probability ofan observed outcome (the cones are indicative of a construction zone onthe road) given those parameter values.

In still another example, the computing device may be configured toprocess information relating to the number and locations of the conesthrough a classifier to determine the likelihood. The classifier can mapinput information (e.g., the number and location of the cones) to aclass (e.g., existence of the construction zone). Examples ofclassifiers and classification algorithms are described above withregard to block 304 of the method 300 illustrated in FIG. 3.

As an example, a training computing device may be configured to receivetraining data for a plurality of driving situations of a given vehicle.For example, respective training data may include, for each of theplurality of driving situations, respective LIDAR-based informationrelating to a respective 3D point cloud of a respective road. Based onthe respective LIDAR-based information of the respective training data,the computing device may be configured to identify respective cones aswell as determine respective number and locations of the respectivecones. Also, the computing device may be configured to receive positiveor negative indication of respective existence of a respectiveconstruction zone corresponding to the respective training data for eachof the driving situations. Further the training computing device may beconfigured to correlate, for each driving situation, the positive ornegative indication with the respective training data, and determineparameters (e.g., vector of weights for equation 1) of the classifierbased on the correlations for the plurality of driving situations. Theseparameters may be provided to the computing device configured to controlthe vehicle such that as the computing device receives the LIDAR-basedinformation, the computing device may be configured to process theLIDAR-based information through the classifier using the determinedparameters of the classifier to determine the likelihood.

In one example, the likelihood may be qualitative such as “low,”“medium,” “high” or may be numerical such as a number on a scale, forexample. Other examples are possible.

Referring back to FIG. 9, at block 912, the method 900 includesmodifying, using the computing device, a control strategy associatedwith a driving behavior of the vehicle, based on the likelihood of theexistence of the construction zone on the road. The computing device maybe configured to modify or select, based on the determined likelihood ofthe existence of the construction zone, a control strategy comprisingrules for actions that control the vehicle speed to safely maintain adistance with other objects and select a lane that is considered safestgiven road changes due to the existence of the construction zone.Examples of modifying the control strategy based on the likelihood aredescribed above with regard to block 306 of the method 300 illustratedin FIG. 3.

At block 914, the method 900 includes controlling, using the computingdevice, the vehicle based on the modified control strategy. Controllingthe vehicle may include adjusting translational velocity, or rotationalvelocity, or both, of the vehicle based on the modified drivingbehavior. Examples of controlling the vehicle based on the modifiedcontrol strategy are described above with regard to block 308 of themethod 300 illustrated in FIG. 3.

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a computer-readable storagemedia in a machine-readable format, or on other non-transitory media orarticles of manufacture. FIG. 11 is a schematic illustrating aconceptual partial view of an example computer program product 1100 thatincludes a computer program for executing a computer process on acomputing device, arranged according to at least some embodimentspresented herein. In one embodiment, the example computer programproduct 1100 is provided using a signal bearing medium 1101. The signalbearing medium 1101 may include one or more program instructions 1102that, when executed by one or more processors may provide functionalityor portions of the functionality described above with respect to FIGS.1-10. Thus, for example, referring to the embodiments shown in FIGS. 3,5, 7, and 9, one or more features of blocks 302-308, 502-512, 702-712,and 902-914 may be undertaken by one or more instructions associatedwith the signal bearing medium 1101. In addition, the programinstructions 1102 in FIG. 11 describe example instructions as well.

In some examples, the signal bearing medium 1101 may encompass acomputer-readable medium 1103, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, etc. In some implementations, the signal bearing medium 1101 mayencompass a computer recordable medium 1104, such as, but not limitedto, memory, read/write (R/W) CDs, R/W DVDs, etc. In someimplementations, the signal bearing medium 1101 may encompass acommunications medium 1105, such as, but not limited to, a digitaland/or an analog communication medium (e.g., a fiber optic cable, awaveguide, a wired communications link, a wireless communication link,etc.). Thus, for example, the signal bearing medium 1101 may be conveyedby a wireless form of the communications medium 1105 (e.g., a wirelesscommunications medium conforming to the IEEE 802.11 standard or othertransmission protocol).

The one or more programming instructions 1102 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as the computing device described withrespect to FIGS. 1-10 may be configured to provide various operations,functions, or actions in response to the programming instructions 1102conveyed to the computing device by one or more of the computer readablemedium 1103, the computer recordable medium 1104, and/or thecommunications medium 1105. It should be understood that arrangementsdescribed herein are for purposes of example only. As such, thoseskilled in the art will appreciate that other arrangements and otherelements (e.g. machines, interfaces, functions, orders, and groupings offunctions, etc.) can be used instead, and some elements may be omittedaltogether according to the desired results. Further, many of theelements that are described are functional entities that may beimplemented as discrete or distributed components or in conjunction withother components, in any suitable combination and location.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

What is claimed is:
 1. A method comprising: receiving, at a computingdevice configured to control a vehicle, from an image-capture devicecoupled to the vehicle, an image of a vicinity of a road on which thevehicle is travelling; selecting, from the image, using the computingdevice, an image portion corresponding to a side of the road at apredetermined height; detecting, using the computing device, aconstruction zone sign in the image portion; determining, using thecomputing device, a type of the construction zone sign; detecting one ormore construction zone objects on the road; determining, based on anumber and locations of the one or more construction zone objects, aseverity of road changes; and controlling, using the computing device,the vehicle based on (i) the type of the construction zone sign, and(ii) the severity of road changes.
 2. The method of claim 1, whereincontrolling the vehicle comprises controlling the vehicle in anautonomous operation mode.
 3. The method of claim 1, wherein detectingthe construction zone sign comprises: detecting a candidate constructionzone sign in the image portion; and determining that the candidateconstruction zone sign relates to a construction zone based on one ormore of shape, color, and pattern of the candidate construction zonesign.
 4. The method of claim 1, further comprising: detecting acandidate construction zone sign in the image portion; determining afirst likelihood that the candidate construction zone sign relates to aconstruction zone based on one or more of shape, color, and pattern ofthe candidate construction zone sign; receiving, at the computingdevice, from a light detection and ranging (LIDAR) device coupled to thevehicle, LIDAR-based information comprising a three-dimensional (3D)point cloud corresponding to the image portion, wherein the 3D pointcloud comprises a set of points based on light emitted from the LIDARand reflected from a surface of the candidate construction zone sign;and determining a second likelihood that the candidate construction zonesign relates to the construction zone based on the LIDAR-basedinformation, wherein detecting the construction zone sign is based onthe first likelihood and the second likelihood.
 5. The method of claim1, further comprising: detecting a candidate construction zone sign inthe image portion; determining a first likelihood that the candidateconstruction zone sign relates to a construction zone, based on one ormore of shape, color, and pattern of the candidate construction zonesign; receiving, at the computing device, from a radio detection andranging (RADAR) device coupled to the vehicle, RADAR-based informationrelating to location and one or more characteristics of the candidateconstruction zone sign; and determining a second likelihood that thecandidate construction zone sign relates to the construction zone basedon the RADAR-based information, wherein detecting the construction zonesign is based on the first likelihood and the second likelihood.
 6. Themethod of claim 5, wherein the RADAR-based information includesdimensional characteristics of the candidate construction zone sign andindicates that the candidate construction zone sign is stationary. 7.The method of claim 1, wherein determining the type of the constructionzone sign comprises: determining a shape of the construction zone sign;and matching the shape to one or more of shapes of typical constructionzone signs.
 8. The method of claim 1, wherein the one or moreconstruction zone objects comprise a cone, a barrel, a worker, abarrier, or construction equipment.
 9. A non-transitory computerreadable medium having stored thereon instructions that, when executedby a computing device of a vehicle, cause the computing device toperform operations comprising: receiving, from an image-capture devicecoupled to the vehicle, an image of a vicinity of a road of travel ofthe vehicle; selecting, from the image, an image portion correspondingto a side of the road; detecting a construction zone sign in the imageportion; determining a type of the construction zone sign; detecting oneor more construction zone objects on the road; determining, based on anumber and locations of the one or more construction zone objects, aseverity of road changes; and controlling the vehicle based on (i) thetype of the construction zone sign, and (ii) the severity of roadchanges.
 10. The non-transitory computer readable medium of claim 9,wherein the one or more construction zone objects comprise a cone, abarrel, a worker, a barrier, or construction equipment.
 11. Thenon-transitory computer readable medium of claim 9, wherein theoperations further comprise: identifying a lane boundary defined by theone or more constructions zone objects, wherein determining the severityof road changes is further based on the identified lane boundary. 12.The non-transitory computer readable medium of claim 9, wherein theoperations further comprise: detecting a candidate construction zonesign in the image portion; determining a first likelihood that thecandidate construction zone sign relates to a construction zone, basedon one or more of shape, color, and pattern of the candidateconstruction zone sign; receiving, from a light detection and ranging(LIDAR) device coupled to the vehicle, LIDAR-based informationcomprising a three-dimensional (3D) point cloud corresponding to theimage portion, wherein the 3D point cloud comprises a set of pointsbased on light emitted from the LIDAR and reflected from a surface ofthe candidate construction zone sign; and determining a secondlikelihood that the candidate construction zone sign relates to theconstruction zone based on the LIDAR-based information, whereindetecting the construction zone sign is based on the first likelihoodand the second likelihood.
 13. The non-transitory computer readablemedium of claim 9, wherein the operations further comprise: detecting acandidate construction zone sign in the image portion; determining afirst likelihood that the candidate construction zone sign relates to aconstruction zone, based on one or more of shape, color, and pattern ofthe candidate construction zone sign; receiving from a radio detectionand ranging (RADAR) device coupled to the vehicle, RADAR-basedinformation relating to location and one or more characteristics of thecandidate construction zone sign; and determining a second likelihoodthat the candidate construction zone sign relates to the constructionzone based on the RADAR-based information, wherein detecting theconstruction zone sign is based on the first likelihood and the secondlikelihood.
 14. The non-transitory computer readable medium of claim 13,wherein the RADAR-based information includes dimensional characteristicsof the candidate construction zone sign and indicates that the candidateconstruction zone sign is stationary.
 15. The non-transitory computerreadable medium of claim 9, wherein determining the type of theconstruction zone sign comprises: determining a shape of theconstruction zone sign in the image portion; and matching the shape toone or more of shapes of typical construction zone signs.
 16. A controlsystem for a vehicle, comprising: an image-capture device; a computingdevice in communication with the image-capture device; and data storagecomprising instructions that, when executed by the computing device,cause the control system to perform operations comprising: receiving,from an image-capture device coupled to the vehicle, an image of avicinity of a road of travel of the vehicle, selecting, from the image,an image portion corresponding to a side of the road; detecting aconstruction zone sign in the image portion, determining a type of theconstruction zone sign, detecting one or more construction zone objectson the road, determining, based on a number and locations of the one ormore construction zone objects, a severity of road changes, andcontrolling the vehicle based on (i) the type of the construction zonesign, and (ii) the severity of road changes.
 17. The control system ofclaim 16, wherein controlling the vehicle comprises controlling thevehicle in an autonomous operation mode.
 18. The control system of claim16, wherein detecting the construction zone sign comprises: detecting acandidate construction zone sign in the image portion; and determiningthat the candidate construction zone sign relates to a construction zonebased on one or more of shape, color, and pattern of the candidateconstruction zone sign.
 19. The control system of claim 16, wherein theone or more construction zone objects comprise a cone, a barrel, aworker, a barrier, or construction equipment.
 20. The control system ofclaim 16, wherein the operations further comprise: identifying a laneboundary defined by the one or more constructions zone objects, whereindetermining the severity of road changes is further based on theidentified lane boundary.